Graphical user interface, system and method for implementing a game controller on a touch-screen device

ABSTRACT

A touch-screen apparatus and method are described for controlling an online video game. For example, one embodiment of a touch-screen apparatus comprises: a touch-screen display for displaying images and receiving user input in response to a user touching the touch screen display; a network interface for establishing a network connection with a gaming server executing a video game in response to user input from the apparatus, the gaming server compressing video output from the video game to generate interactive compressed streaming video, and transmitting the interactive compressed streaming video generated by the video game over the network connection to the touch-screen apparatus; a memory for storing program code and a processor for processing the program code to generate a touch-screen graphical user interface (GUI) comprising: a plurality of user input elements providing user input in response to the user selecting the user input elements on the touch screen display; wherein the user input is transmitted from the touch-screen apparatus to the gaming server to control the execution of the video game.

CLAIM OF PRIORITY

The application is a continuation-in-part and claims the benefit of U.S.patent application Ser. No. 13/016,785 entitled “Graphical UserInterface, System and Method For Implementing A Game Controller On ATouch-Screen Device”, filed on Jan. 28, 2011 which is acontinuation-in-part and claims the benefit of U.S. ProvisionalApplication No. 61/351,268 entitled, “Graphical User Interface, Systemand Method For Implementing A Game Controller On A Touch-Screen Device”,filed on Jun. 3, 2010.

TECHNICAL FIELD

The present disclosure relates generally to the field of data processingsystems and, more particularly, to a graphical user interface forcontrolling a game on a touch-screen device.

BRIEF DESCRIPTION OF THE DRAWINGS

A better understanding of the present invention can be obtained from thefollowing detailed description in conjunction with the followingdrawings, in which:

FIGS. 1 a-b illustrate a game controller.

FIG. 2 illustrates a graphical user interface (GUI) for implementing agame controller on a touch screen device.

FIG. 3 illustrates a framework for implementing a touch-screen userinterface on a touch screen device and a hosting service providingrendering commands and video to the touch-screen device.

DESCRIPTION OF EMBODIMENTS OF THE INVENTION

The assignee of the present application has developed an online videogaming system. Certain embodiments of this system are described, forexample, in U.S. patent application Ser. No. 12/538,081, filed, Aug. 7,2009, entitled, “System and Method for Compressing Video Based onLatency Measurements and Other Feedback” and U.S. application Ser. No.12/359,150, filed Jan. 23, 2009, entitled, “System And Method forProtecting Certain Types of Multimedia Data Transmitted Over ACommunication Channel.” These applications are sometimes referred toherein as the “co-pending applications” and are incorporated herein byreference.

Described herein is a unique controller and touch-screen graphical userinterface (GUI) for controlling online video games as described in theco-pending applications. While the controller and touch screen GUIdescribed below may be used to control “online” games in one embodimentof the invention, the underlying principles of the invention are notlimited to “online” games. For example, the controller and touch screenGUI described below may be used to control games being executed locallyon the gaming apparatus to which they are connected (in the case of thecontroller) and/or on which they are displayed (in the case of the GUI).

As illustrated in FIGS. 1 a-b, one embodiment of a physical controllerincludes left and right joysticks 101 and 102, respectively, which maybe freely moved in all directions to perform game input functions. Forexample, in an adventure game or first person shooter, the joysticks maybe manipulated to move a character throughout the game. Also shown inFIG. 1 is a directional pad (commonly referred to as a D-pad) 108 toperform up, down, left, and right motion and/or cursor controls. A groupof four action buttons 105 are provided to perform variousapplication-specific functions (e.g., jump, run, switch weapons, etc). Aset of transport buttons 104 positioned towards the bottom of thecontroller allow the user to perform play/pause, stop, fast forward, andrewind operations from within a game (e.g., to pause, fast forward,rewind, etc, within the game). A set of three navigation buttons 103provided towards the middle of the controller may provide variousdifferent navigation functions and/or other system-specific functions.For example, in one embodiment, the central circular button causes auser navigation interface to appear overlaid on top of the current game;the left button is used to back out of menu items; and the right buttonis used to enter/select menu items. Of course, the underlying principlesof the invention are not limited to any particular set of buttonfunctions. Bumper buttons 106 a-b (sometimes referred to as shoulderbuttons) and trigger buttons 107 a-b provide various well-known gamespecific functions (e.g., fire, change weapon, reload, etc).

As illustrated in FIG. 2, one embodiment of a touch screen graphicaluser interface (GUI) includes a set of graphical buttons which visuallycorrespond to the buttons on the physical controller illustrated inFIG. 1. Thus, using the embodiments described herein, a user who isfamiliar with the layout of the physical controller will find itrelatively easy to identify the appropriate buttons and play a game on atouch screen device. In one embodiment, the touch screen device is anApple iPad. However, the underlying principles of the invention may beimplemented on any type of touch screen device (e.g., an Apple iPhone,Apple iPod, a Google Android, a Microsoft Windows, a Palm or Research inMotion Blackberry touch screen device, etc).

The numerical designations in FIG. 2 correspond to the numericaldesignations used in FIG. 1. For example, graphical joysticks 201-202correspond to physical joysticks 101-102; the graphical directional pad208 corresponds to the physical directional pad 108; the graphic actionbuttons 205 correspond to physical action buttons 105; the graphicalnavigation buttons 203 correspond to physical navigation buttons 103;the graphical bumper buttons 206 a-b correspond to physical bumperbuttons 106 a-b; and the graphical trigger buttons 207 a-b correspond tophysical trigger buttons 107 a-b. Unless otherwise stated herein, thegraphical buttons perform the same functions as the correspondingphysical buttons on the physical controller.

In one embodiment, touching the joysticks 201-202 indicates to the touchscreen device that the user is intending to manipulate the joysticks,and subsequent motion while continuously touching the joystick 201-202is interpreted by the touch screen device as if the user had moved aphysical joystick a similar distance from the center of the joystick. Inone embodiment, a touch that is closer to the center of the joystickthan to any other button is considered to the user touching the joystickand activating it, and a touch closer to another button is considered atouch of the other button. In one embodiment, a touch that activates thejoystick is considered to define the center point position of thejoystick, so that any subsequent motion from the center point isconsidered to be a movement away that center point. In anotherembodiment, a touch that activates the joystick is considered to havemoved the joystick to the position of this first touch, and subsequentmotion from this point is considered to have moved the joystick fromthat position. In one embodiment, there is an option where either theuser or the game being played is able to select whether the first touchis interpreted as touching the joystick at its center point or movingthe joystick to the position of the touch. The physical joysticks 101and 102 may be equipped with a press down button capability, such thatif the user presses down on a joystick, that downward press is detected.In one embodiment, the joysticks 201 and 202 are responsive to a “doubletap”, in which a rapid touch-touch action (with a limited time durationbetween the taps defined by the touch screen device, the game, or set byuser as a setting, so as to distinguish a “double tap” from a release ofthe joystick followed by a retouch by the finger) is defined to beanalogous to pressing down the physical joysticks 101 and 102. In oneembodiment, such a “double tap” is only interpreted as such within alimited radius of the center of the joystick 201 or 202.

In one embodiment, touch of the buttons in FIG. 2 other than thejoystick is interpreted as a press of the analogous button on thephysical gamepad in FIG. 1. In one embodiment, if a button is pressedfollowed by a drag operation, then the drag is interpreted as anactuation ramp of a variable value. For example, on the physical gamepadin FIG. 1 the 107 b and 107 a (“Left Trigger” or “LT” and “RightTrigger” or “RT”) buttons may be variable analog controls that may beused for purposes such as controlling the degree of acceleration of acar in a car game. If the user presses LT 207 b or RT 207 a, and thendrags away from those buttons, the degree of drag would be interpretedas if the user had pressed button 107 b or 107 a to variable degree ofactuation. In one embodiment, when the user drags after pressing 207 bor 207 a, a graphic gauge appears behind or near the button to indicateto the user the degree of actuation of the variable controller. Alsoillustrated in FIG. 2 is a graphical keyboard button which, whenselected by the end user, causes a graphical keyboard to be displayedfor entering text. In one embodiment, the online gaming servicedescribed in the co-pending applications communicates with thetouch-screen device to automatically cause the graphical keyboard toappear when text entry is needed. Moreover, if the user touches a textentry field from within a game (or other application) the graphicalkeyboard may automatically appear.

A graphical mouse button 211 illustrated in FIG. 2 allows the user tomove throughout the game as if the user of the touch-screen device wereusing a mouse. For example, in one embodiment, if the user touches anddrags a thumb across the display screen on which the video game isdisplayed, a corresponding mouse control signal is transmitted from thetouch-screen device to the online gaming service (e.g., dragging a thumbto the left causes a left mouse scroll operation). In one embodiment,selecting the mouse button 211 causes the button to toggle to atouch-screen button (e.g., indicated in the form of a graphical hand),thereby allowing the user to implement various functions specific totouch-screen technology (e.g., double-tap to select, double drag to zoomin/out, etc). In response, the touch-screen device will transmittouch-screen specific control signals to the online gaming service whichwill responsively control the current video game and transmit theresulting video stream to the touch-screen device.

A graphical record button 212 is also provided. In one embodiment, whenselected, the record button causes the online gaming service to beginrecording the user's game video output, or game actions. The recordedgame video or game actions may then be used for many purposes asdescribed in the co-pending applications, including Brag Clips™ videorecordings, which, which are recordings of previous game play may bereviewed by the user and/or shared with others.

The button layout and functionality illustrated in FIG. 2 is beneficialfor a variety of reasons. First, because the buttons are positioned atthe top of the display (as opposed to the bottom), the user may usehis/her thumbs to select the buttons and control the game while theremainder of the user's fingers are positioned under the touch-paddevice (e.g., to hold the device securely during gameplay). Other thanbuttons 107 a-b and 103 in FIG. 1, the other buttons and switches areusually controlled by the thumbs (and even 103 are sometimes controlledby the thumbs) and, so analogous thumb control of these buttons on atouch screen could be more familiar to a user than control by otherfingers

In addition, in one embodiment, when a user is actively controlling thejoystick controls 201-202, the buttons 208, 205 surrounding the joystickcontrols and potentially other touch sensitive areas are “deactivated”while the user continues to manipulate the joystick. Thus, after theuser touches the touch screen over the image of the joystick 201 or 202,and does not cease to touch the touch screen with that finger whilemoving the finger around, the touch screen device will consider that thejoystick is still being manipulated, and despite the fact the finger maypass over another button, the touch screen device will not interpretthat as a touch of the button underneath the finger, until the finger islifted from the touch screen and again touches the touch screen. Thisallows the user to touch the joystick 201 or 202 and then have a widerarea of motion to manipulate the joystick than would normally bepossible if the motion were constrained to avoid moving over a button205 or 208, resulting in the erroneous interpretation as a touch of thatbutton. Also, given the intensity of many games, it allows the user tovigorously manipulate the joystick without fear of inadvertently hittinga nearby button, which might result in an undesired behavior in the game(e.g. inadvertently shooting an ally while using the joystick to turnquickly with an extreme motion).

In one embodiment, the range of that the user will be allowed tomanipulate the joystick is limited to some specified distance from thecenter of the joystick, for example, to a circle of some radius aroundthe joystick. The game could either cease to interpret the motion oncethe range of motion is exceeded (and the user could potentially realizebecause there was no corresponding action in the game beyond theallowable range of motion that the range had been exceeded), or therecould be some other indication to the user, such as an auditory alert(e.g. a beep) or a graphical indication (e.g. a flash on the screen).

In one embodiment, when a joystick 201 or 202 is interpreted asactivated, the nearby buttons are dimmed to be less prominent than theyare usually (or some other graphical indication is used) to remind theuser that touching the nearby buttons (without first releasing thejoystick) will not be interpreted as a button press. For example, whenthe joystick 201 is activated, the D-pad buttons 208 would be dimmed outor when joystick 202 is activated one of the action buttons 205 would bedimmed out. When the user lifts his/her thumbs up from the joysticks,the D-pad buttons 208 or one of the action buttons 205 would becomeactive again and would be restored to their normal appearance, providinga visual indication to the user that they can be actuated. In oneembodiment, when the user lifts his/her fingers off of the graphicaljoysticks 201-202 as described, the graphical joysticks may beconfigured to be interpreted as (a) remaining in the last position theywere in or (b) returning to a center position. In one embodiment, suchstate in the preceding sentence would be shown visually by the positionthat the joystick 201 or 202 graphical image is draw in.

Similarly, to the joystick 201 and 202 actuation being handled overnearby buttons in the preceding three paragraphs, dragging afterpressing a button such as the LT 207 b and RT 207 a button could, in oneembodiment, deactivate nearby buttons to allow motion that overlapsnearby buttons. Similar to the joystick 201 and 202 actuation, suchdragging can be limited in range, and visual and/or auditory indicatorscan be provided to the user.

As mentioned above, in one embodiment the GUI shown in FIG. 2 is used toplay remotely-operated games from a gaming service as described in theco-pending applications. In one embodiment, the communication with theonline gaming service may occur over various network links includingwireless links (e.g., digital 3G or 4G cellular links, Wi-Fi links,Bluetooth links, etc) and wired links (e.g., Ethernet, USB, etc).

In one embodiment, the game that is executing in the touch screen deviceor in a remotely-operated game requests the particular configuration ofbuttons so as to best suit the needs of the game.

In one embodiment, and non-game application is used, and it requests aninterface suitable for its operation.

In one embodiment, the touch interface described herein is rendered bythe local touch screen device. In another embodiment, the touchinterface described herein is rendered by a remotely-operating game orapplication such as that described in the co-pending applications. Inanother embodiment part of the touch interface described herein isrendered locally and part is rendered remotely.

As illustrated in FIG. 3, in one embodiment, the touch-screen device 303includes a framework 315 which renders different GUI elements 316 (alsocalled widgets) as a function of what the hosting service 310 instructsthe touch-screen device to render via rendering commands 301. Theframework 315 also allows the user to associate custom actions to betransmitted back to the touch screen device 303 as per instructionsfetched by the remote device 303 itself. In practice, the frameworkallows a remote video game hosting service 310 to instruct the touchdevice 303 to render a button (e.g., button “A”) in a specific positionon the screen and potentially override the behavior of that button tosend a custom action back to the hosting service 310 itself.

In one embodiment, as the hosting service 310 stores the UIconfiguration using a meta language that specifies the type, size,rotation and location of the UI element to be rendered and the list ofactions to be executed upon user interaction with the rendered UIelement. There could be multiple actions associated with a UI widget asa function of the user interaction. For instance, one UI widget couldsupport different actions being returned to the hosting service 310 for“press” or “swipe” independently.

In one embodiment, the various graphical elements illustrated herein andthe associated functions may be generated by a general purpose or aspecial purpose processor executing instructions. For example, aprocessor within the touch screen device may execute instructions togenerate the graphical buttons shown in FIG. 2; receive user input fromthe touch screen; and responsively provide control signals to the onlinegaming service described in the co-pending applications. Embodiments ofthe invention may also include various steps as set forth above. Thesteps may be embodied in machine-executable instructions which cause ageneral-purpose or special-purpose processor to perform certain steps.Various elements which are not relevant to these underlying principlessuch as computer memory, hard drive, input devices, have been left outof some or all of the figures to avoid obscuring the pertinent aspects.

Elements of the disclosed subject matter may also be provided as amachine-readable medium for storing the machine-executable instructions.The machine-readable medium may include, but is not limited to, flashmemory, optical disks, CD-ROMs, DVD ROMs, RAMs, EPROMs, EEPROMs,magnetic or optical cards, propagation media or other type ofmachine-readable media suitable for storing electronic instructions. Forexample, the present invention may be downloaded as a computer programwhich may be transferred from a remote computer (e.g., a server) to arequesting computer (e.g., a client) by way of data signals embodied ina carrier wave or other propagation medium via a communication link(e.g., a modem or network connection).

It should also be understood that elements of the disclosed subjectmatter may also be provided as a computer program product which mayinclude a machine-readable medium having stored thereon instructionswhich may be used to program a computer (e.g., a processor or otherelectronic device) to perform a sequence of operations. Alternatively,the operations may be performed by a combination of hardware andsoftware. The machine-readable medium may include, but is not limitedto, floppy diskettes, optical disks, CD-ROMs, and magneto-optical disks,ROMs, RAMs, EPROMs, EEPROMs, magnet or optical cards, propagation mediaor other type of media/machine-readable medium suitable for storingelectronic instructions. For example, elements of the disclosed subjectmatter may be downloaded as a computer program product, wherein theprogram may be transferred from a remote computer or electronic deviceto a requesting process by way of data signals embodied in a carrierwave or other propagation medium via a communication link (e.g., a modemor network connection).

Additionally, although the disclosed subject matter has been describedin conjunction with specific embodiments, numerous modifications andalterations are well within the scope of the present disclosure.Accordingly, the specification and drawings are to be regarded in anillustrative rather than a restrictive sense.

1. An touch-screen apparatus comprising: a touch-screen display fordisplaying images and receiving user input in response to a usertouching the touch screen display; a network interface for establishinga network connection with a gaming server executing a video game inresponse to user input from the apparatus, the gaming server compressingvideo output from the video game to generate interactive compressedstreaming video, and transmitting the interactive compressed streamingvideo generated by the video game over the network connection to thetouch-screen apparatus; a memory for storing program code and aprocessor for processing the program code to generate a touch-screengraphical user interface (GUI) comprising: a plurality of user inputelements providing user input in response to the user selecting the userinput elements on the touch screen display; wherein the user input istransmitted from the touch-screen apparatus to the gaming server tocontrol the execution of the video game.
 2. The apparatus as in claim 1wherein the user input elements include: one or more joystick controlelements for receiving joystick-based input in response to a usertouching the joystick control elements with a finger and dragging thefinger along the surface of the touch screen display; a plurality ofuser input buttons surrounding the joystick control elements, the userinput buttons providing user input in response to the user selecting theuser input buttons on the touch screen display; wherein thejoystick-based input and other user input is transmitted from thetouch-screen apparatus to the gaming server to control the execution ofthe video game.
 3. The apparatus as in claim 2 wherein, in response to auser touching and dragging one of the joystick control elements, theuser input buttons surrounding the selected joystick control elementbecome temporarily deactivated.
 4. The apparatus as in claim 3 whereinthe user input buttons become re-activated in response to the userremoving a finger from the joystick control element.
 5. The apparatus asin claim 2 wherein a center point position of a joystick control elementis defined by a location on the touch-screen display at which thejoystick control element is initially touched by the end user.
 6. Theapparatus as in claim 2 wherein a center point position of a joystickcontrol element is defined in a consistent location on the touch-screendisplay regardless of the point at which the joystick control element isinitially touched by the end user.
 7. The apparatus as in claim 2wherein a double-tap on a joystick control element by the end userresults in user input indicating that the user has pressed downward onthe joystick control element.
 8. The apparatus as in claim 2 wherein theGUI comprises a left joystick control element displayed in a left regionof the touch-screen display and a right control element displayed in aright region of the touch screen display.
 9. The apparatus as in claim 8wherein the user input buttons include a left trigger button displayedin the left region of the touch-screen display and a right triggerbutton displayed in the right region of the touch-screen display. 10.The apparatus as in claim 9 wherein in response to a user selecting theleft trigger button or the right trigger button on the touch-screendisplay with a finger and dragging the finger, the resulting user inputindicates that the user has actuated the left trigger button or theright trigger button by an amount proportional to the distance overwhich the finger is dragged.
 11. The apparatus as in claim 10 whereinthe GUI comprises a graphical gauge displayed behind or near the righttrigger button or left trigger button to indicate to the user the degreeof actuation.
 12. The apparatus as in claim 2 wherein the GUI includes amouse control mode in which, in response to a user touching and draggingat a random location of the touch-screen display, user input isgenerated to indicate a mouse movement in a direction corresponding tothe direction of the dragging motion.
 13. The apparatus as in claim 2wherein one of the user input buttons comprises a record button causingthe video game server to begin recording the interactive compressedstreaming video.
 14. The apparatus as in claim 3 wherein the user inputbuttons surrounding the joystick control element become dimmed to remindthe user that the user input buttons are temporarily deactivated. 15.The apparatus as in claim 2 wherein in response to the user dragging oneof the joystick control elements and then lifting a finger from thetouch screen display, the resulting user input is interpreted to causethe joystick control element to remain in its current position prior tothe user lifting the finger.
 16. The apparatus as in claim 2 wherein inresponse to the user dragging one of the joystick control elements andthen lifting a finger from the touch screen display, the resulting userinput is interpreted to cause the joystick control element to return toa center position.
 17. A machine-readable medium having program codestored thereon to be executed on a touch-screen apparatus having atouch-screen display for displaying images and receiving user input inresponse to a user touching the touch screen display, a networkinterface for establishing a network connection with a gaming serverexecuting a video game in response to user input from the touch-screenapparatus, the gaming server compressing video output from the videogame to generate interactive compressed streaming video, andtransmitting the interactive compressed streaming video generated by thevideo game over the network connection to the touch-screen apparatus,wherein program code is executable by the touch-screen apparatus tocause the apparatus to generate a graphical user interface comprising: aplurality of user input elements providing user input in response to theuser selecting the user input elements on the touch screen display;wherein the user input is transmitted from the touch-screen apparatus tothe gaming server to control the execution of the video game.
 18. Themachine-readable medium as in claim 17 wherein user input elementsinclude: one or more joystick control elements for receivingjoystick-based input in response to a user touching the joystick controlelements with a finger and dragging the finger along the surface of thetouch screen display; a plurality of user input buttons surrounding thejoystick control elements, the user input buttons providing user inputin response to the user selecting the user input buttons on the touchscreen display; wherein the joystick-based input and other user input istransmitted from the touch-screen apparatus to the gaming server tocontrol the execution of the video game.
 19. The machine-readable mediumas in claim 18 wherein, in response to a user touching and dragging oneof the joystick control elements, the user input buttons surrounding theselected joystick control element become temporarily deactivated. 20.The machine-readable medium as in claim 19 wherein the user inputbuttons become re-activated in response to the user removing a fingerfrom the joystick control element.
 21. The machine-readable medium as inclaim 18 wherein a center point position of a joystick control elementis defined by a location on the touch-screen display at which thejoystick control element is initially touched by the end user.
 22. Themachine-readable medium as in claim 18 wherein a center point positionof a joystick control element is defined in a consistent location on thetouch-screen display regardless of the point at which the joystickcontrol element is initially touched by the end user.
 23. Themachine-readable medium as in claim 18 wherein a double-tap on ajoystick control element by the end user results in user inputindicating that the user has pressed downward on the joystick controlelement.
 24. The machine-readable medium as in claim 18 wherein the GUIcomprises a left joystick control element displayed in a left region ofthe touch-screen display and a right control element displayed in aright region of the touch screen display.
 25. The machine-readablemedium as in claim 24 wherein the user input buttons include a lefttrigger button displayed in the left region of the touch-screen displayand a right trigger button displayed in the right region of thetouch-screen display.
 26. The machine-readable medium as in claim 25wherein in response to a user selecting the left trigger button or theright trigger button on the touch-screen display with a finger anddragging the finger, the resulting user input indicates that the userhas actuated the left trigger button or the right trigger button by anamount proportional to the distance over which the finger is dragged.27. The machine-readable medium as in claim 26 wherein the GUI comprisesa graphical gauge displayed behind or near the right trigger button orleft trigger button to indicate to the user the degree of actuation. 28.The machine-readable medium as in claim 18 wherein the GUI includes amouse control mode in which, in response to a user touching and draggingat a random location of the touch-screen display, user input isgenerated to indicate a mouse movement in a direction corresponding tothe direction of the dragging motion.
 29. The machine-readable medium asin claim 18 wherein one of the user input buttons comprises a recordbutton causing the video game server to begin recording the interactivecompressed streaming video.
 30. The machine-readable medium as in claim19 wherein the user input buttons surrounding the joystick controlelement become dimmed to remind the user that the user input buttons aretemporarily deactivated.
 31. The machine-readable medium as in claim 18wherein in response to the user dragging one of the joystick controlelements and then lifting a finger from the touch screen display, theresulting user input is interpreted to cause the joystick controlelement to remain in its current position prior to the user lifting thefinger.
 32. The machine-readable medium as in claim 18 wherein inresponse to the user dragging one of the joystick control elements andthen lifting a finger from the touch screen display, the resulting userinput is interpreted to cause the joystick control element to return toa center position.